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1. INTRODUCTION 

Until recently, autonomous vehicles were merely a dream outside research centers at a few universities 
and companies. Thanks to rapid developments in computer processing speed, image processing techniques, 
and miniaturization, this dream has become a reality that one can see in the street [1]. Vehicles are now able to 
sense the world around them and operate autonomously in different environments to perform several tasks in 
addition to driving, including firefighting, mine detection and clearance, and surveillance [2]. To perform such 
missions, reaching the destination point is an essential task. Determining the correct steering angle that will 
lead to the target point is key [3]. 

Most steering angle calculation methods can be categorized into two approaches: computer vision- 
based methods and learning-based methods [4]. The first approach depends on extracting the required features 
to define the lane lines and then performing trigonometric calculations to determine the steering angle. The 
second approach exploits the power of a neural network (NN) to determine the angle. The NN-based methods 
are mainly end-to-end, with the system receiving the input video frame with the related steering angle in each 
frame and feeding it into a convolutional neural network (CNN), which predicts the steering angle based on 
the weights previously obtained by training the NN [5]. 

In this work, we exploit the advantages of both approaches and propose an algorithm that has the 
accuracy and generalization of the learning-based methods and the low computational cost of the vision-based 
methods. The proposed system receives the coordinates of points on the lane lines and tests if one or more lines 
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were detected. Based on the number of detected lines, the system determines the path line that the vehicle is 

following, then calculates an equation that represents that line. The coefficients of the equation are fed into the 

trained NN, which predicts the steering angle accordingly. The main contributions of this study are as: 

- Different types of lane detection algorithms are used to provide lane point coordinates to the system. This 
is not limited to NN-based lane detection approaches, but can also exploit the traditional lane and edge 
detection algorithms. 

- The path line followed by the vehicle is determined efficiently and represented by a quadratic equation. 

- An accurate steering angle is robustly predicted at a high speed, measured in frames per second (FPS). 

We now review recent studies in the field of steering angle prediction, following the aforementioned 
classification from [4]. The recent studies on each of the two approaches are presented, and in section 3, their 
results are compared with the proposed method. 

On the one hand, from a learning-based method point of view, most algorithms use the end-to-end 
approach for steering angle prediction. Some of them use simulators, as is the case in [6], [7], while others, 
such as [8], [9], use custom hardware that is trained and tested in special environments. The results of such 
methods depend on the driving style in the training phase, which makes it hard to compare their results with 
other studies. Using standard benchmarks to test a proposed algorithm can make it easier to evaluate against 
other available approaches. 

Du et al. [10] proposed two different end-to-end algorithms, one based on a transfer learning model 
using ResNet50, and the other using a 3D CNN with long short-term memory (LSTM). They used the Udacity 
dataset [11] to test their algorithms. Acceptable results were recorded in this work, but substantial 
computational resources were required. Wu et al. [12] presented another method using the LSTM idea: an end- 
to-end system depending on a convolutional-LSTM NN with a multi-scale spatiotemporal integration (MSI) 
module. Their experiments achieved acceptable results on the Udacity benchmark, but the system failed to 
control the steering angle smoothly. 

Three different systems were presented in [13], one based on RGB images, one using a segmentation 
scheme, and one using a depth image to predict the steering angle. A combination of these methods was used 
to improve the results and determine the features required to obtain accurate results. Promising results were 
registered at the cost of expensive computation in which the processed data were fed to the “super” network 
that learns the important features from each network. 

Hou et al. [14] exploited heterogeneous auxiliary network feature mimicking to create an accurate 
and robust steering control system. The authors found that this reduces the run time by discarding the auxiliary 
networks after the training phase. However, this limits the generalization ability. A method called Feudal 
Steering was presented in [15]. Feudal networks were utilized in hierarchical reinforcement learning (HRL) to 
determine vehicles’ steering angles. In this approach, the task was divided into manager and worker sub- 
networks to efficiently produce accurate results. 

Kim and Canny [16] proposed a new steering prediction algorithm that consists of two stages. In the 
first stage, a visual attention model is used to train a CNN. The attention model is used to indicate the important 
regions in the image that affect the output of the network. In the second stage, a causal filter is applied to find 
the region in the image that has an actual influence on the prediction of the steering angle. The proposed 
approach was tested on the Udacity and Comma.ai [17] benchmarks and promising results were recorded. 

Chen [18] introduced a simple end-to-end steering control algorithm for self-driving cars, consisting 
of a CNN that was trained and tested on the Comma.ai dataset. The system obtains an image from a front view 
camera and feeds it to the CNN for steering angle prediction. Such schemes produce acceptable speed and 
robustness. 

On the other hand, computer vision-based paradigms generally have two phases: lane line feature 
extraction and steering angle calculation. The first step may use traditional lane detection techniques, such as 
edge detection or the Hough transform [19], or it may utilize an NN-based lane detection approach [20]. 

Rochan et al. [21] proposed an algorithm for steering angle calculation that consists of three phases. 
Phase one uses a Gaussian mixture model and an expectation maximizing approach to extract the drivable road 
parts. Next, the steering angle is calculated as the deviation of the intersection point of the roads’ edges from the 
midline of the image. In the last phase, spurious steering angle transition noise is removed using a Kalman filter. 

Another approach was proposed by Rasib et al. in [22], which has two main stages. First, pixel-level 
segmentation is performed to extract and indicate the road region in the image using a CNN end-to-end scheme. 
Then, trigonometric and line interception are used to compute the steering angle. The authors also presented a 
new benchmark named carl-dataset that can be used to test and evaluate steering angle prediction algorithms. 

A comparison study was performed by Nguyen et al. in [23] to evaluate the performance of a vision- 
based method compared with an advanced deep learning method. They presented two methods, the first one 
based on standard image processing methods such as edge detection and the Hough transform, and the second 
one based on a CNN. The first method has four steps: pre-processing, edge detection, the Hough transform, 
and finally steering angle computation. The second method is an end-to-end CNN-based approach. Testing 
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both methods on a common dataset showed that the image processing-based method is faster and attains more 
accurate results in static and simple backgrounds. Nevertheless, the CNN-based method was better generalized 
to the changes in the background. 


2. METHOD 

Since an autonomous land vehicle in a neural network (ALVINN) [24] successfully used neural 
networks to drive a vehicle autonomously, a variety of studies have applied NNs and introduced new self- 
driving algorithms [25]. Unlike the models presented in the literature, the proposed method treats steering angle 
prediction as a regression problem in a novel manner. The following pseudo-code summarizes the main ideas 
of the Algorithm 1. 


Algorithm 1. Pseudo-code summarizes the main ideas 
Start 
Video frame received with the lane line points detected previously. 
Calculate the lane line equation(s) from the detected points. 
If: both lane lines are detected: then 
Calculate the path line between them 
Feed the path line equation to the NN 
Steering angle value = output of NN 


2.1. Getting points on the lane line 

The proposed algorithm needs to determine the lane feature coordinates in the incoming frame for use 
later in the steering angle prediction process. This information is obtained using lane detection algorithms like 
CurveLane-NAS [26], E2E-LMD [27], or LaneATT [28]. This information is used later to calculate the 
quadratic equations representing the lane lines. This step adds great flexibility by allowing the proposed system 
to adapt to different lane line detection algorithms. 


2.2. Calculating lane line equations 

The feature coordinates are then used to find one or more equations representing the lane lines. If both 
left and right lane lines were detected in the last phase, the received coordinates of the feature are categorized into 
two groups indicating each lane line. All the points in each group are fitted to build a quadratic equation curve 
that represents that lane line. Then a center curve between both lane curves is calculated, giving the path line: 


_ Xright—*left 
Xcenter = + Xleft (1) 


2 
since Viert = YVrignt> Ycenter takes the same value. Figure | explains this step and shows the incoming and 
resulting lane lines. 

Figure 1 illustrates the case in which both lane lines are detected: Figure 1(a) shows that both left and 
right lanes (blue and orange respectively) are detected, while Figure 1(b) shows the center curve path denoted 
by the green dotted line. The center path is the path that the vehicle will follow. In the case in which only one 
lane line (left or right) is detected, the system calculates the equation for that line and considers it as the path 
line. After the path line equation is determined, it is fed into the neural network for steering angle prediction. 


Left Line 
Right Line 
— — P Center Path 


Left Line 
Right Line 


(a) (b) 


Figure 1. The case in which both lines are detected (a) left and right lane lines and (b) both lines and the new 
calculated path line 
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2.3. Network architecture 

The Keras library [29] was used to build the artificial neural network (ANN) deployed in the proposed 
algorithm. The proposed ANN consists of 6 dense layers. The first 5 dense layers each have a rectified linear 
unit (ReLU) activation function and are each followed by a dropout layer. The first layer has a depth of 1024 
with a dropout of 40%, and the second has a depth of 512 and a dropout of 30%. The depth of the following 
two layers is 256 with dropouts of 20% and 10%, respectively. The fifth dense layer has a depth of 128 with a 
5% dropout. The last dense layer is a single neuron that has a linear activation function with one output, which 
represents the predicted steering angle. Figure 2 illustrates the architecture of the ANN used in this work. 


=, => => a) 
1 Dense 
and 
Linear 
1024 Dense 512 Dense 256 Dense 256 Dense 128 Dense 
and and and and and 
ReLU ReLU ReLU ReLU ReLU 


Figure 2. The architecture of the proposed ANN 


2.4. Loss function and optimization 

The mean square error (MSE) loss function was used to train the model. Due to the square term, this 
function harshly penalizes large errors, which must be avoided. MSE is a common loss function used in regression 
problems [10]. It is defined as the average of all square differences between real and predicted values: 


MSE =—YN (yi — 91)? (2) 


where y; represents the predicted steering angle, ¥, indicates the real steering angle, and N is the total number 
of predicted values. The loss was optimized using the Adam optimizer [30], which is acommon choice in ANN 
deep learning applications as it usually outperforms other stochastic gradient descent methods because of its 
computational efficiency and high adaptive learning ability [13]. 


2.5. Training 
Before starting the training process, the dataset was tested to check the distribution of the steering 


angle values. It is important to have a balanced dataset in the training process to prevent any bias [31]. As 
shown in Figure 3 shows the normalized steering angle distribution of the dataset from Comma.ai [17]. 
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Figure 3. The normalized steering angle distribution of the training dataset 
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Figure 3 indicates that the dataset is fairly distributed over the range of steering angle values, with a 
near-Gaussian distribution except for a sharp peak around zero, as the driver keeps the steering wheel in the 
neutral position most of the time. The dataset is therefore suitable to train the system. The dataset was split 
with a 7:3 ratio, with 70% (771) for training and the remaining 30% (329) for validation. After the training 
process started, the model began to learn how to predict the steering angle based on the available line curve 
equation coefficients. Figure 4 shows the model loss during training and validation. It can be seen from Figure 
4 that the system started to converge after around 25 epochs of iterative training. 
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Figure 4. The training and validation loss curves 


2.6. Steering angle prediction 

The proposed method estimates the steering angle of the autonomous vehicle using the path line 
equation coefficients. So, to operate the proposed algorithm, the coefficients of the path line equation obtained 
during the last phase are introduced to the first dense layer, with the connection weights changing accordingly. 
The outputs from the first layer are fed to the next layer and so on until the final neuron produces the predicted 
steering angle. 


2.7, Evaluation metrics 

To evaluate the proposed method’s performance, the mean absolute error (MAE) and the root mean 
square error (RMSE) were used. The MAE, defined by (3), gives all predictions the same weight to calculate 
the average error. The RMSE, defined by (4), is the standard deviation of the tested data and also provides a 
unit-based evaluation of the system [13]. The prevalence of the predicted data refers to the accuracy of the 
tested system. A low RMSE value indicates high accuracy. 


MAE =~ SN alyi— Fil (3) 


RMSE = [-Y8.04- 9)? (4) 


The MAE is always less than the RMSE; the difference indicates the variance in the predictions. Lower 
difference values indicate low variance, while higher values indicate high variance [13]. 


3. RESULTS AND DISCUSSION 

Extensive experiments have been conducted on various datasets to test and evaluate the proposed 
method. To evaluate the proposed system fairly, the results of those experiments were compared to some recent 
studies’ results. The results for each benchmark are presented separately, followed by a detailed discussion. 


3.1. Datasets 

The Udacity [11] and Comma.ai [17] benchmarks were used to test and evaluate the proposed 
algorithm. The Udacity dataset has 8 hours of driving data in sunny and overcast weather, containing several 
challenging scenes. Comma.ai has 11 video clips with a total of 7.25 hours of driving data. Further details 
about the datasets are given in Table 1. 
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Table 1. Details of the datasets used for testing 


Dataset No. of frames FPS Duration (hours) Lighting Road 
Udacity 577,900 20 8 Day Urban 
Comma.ai 522,434 20 7.25 Day and Night _ Highways and Urban 


Both datasets were used separately. During the tests, each dataset was processed and, as explained in 
section 2.2, a table containing the lane line information for the current video frame was fed into the network. 
Then, the system predicted the steering angle for that frame, and the process was repeated for each frame. 


3.2. Experimental results 

Extensive tests were performed on the two datasets to evaluate the proposed algorithm and compare 
the results with those of some recent studies, namely [10], [12]-[16], [18]. First, the scheme was tested on the 
Udacity dataset and the results were compared with other models that have been applied to the steering angle 
prediction task. The results are listed in Table 2. 


Table 2. Comparison of the proposed algorithm with existing methods on the Udacity dataset 


Method [ref.] MAE (degrees) | RMSE (degrees) 
3D LSTM [10] - 6.44 
ResNet50 Transfer [10] - 4.06 
RGB + Seg [13] 1.19 2.43 
RGB + Depth [13] 1.02 2.32 
RGB + Seg + Depth [13] 1.25 2.56 
FM-Net [14] 1.62 2.35 
Feudal Steering [15] 1.09 2.67 
MSINet [12] - 2.81 
CNN + FCN [16] 4.12 4.83 
CNN + LSTM [16] 4.15 4.93 
Ours 1.04 2.33 


Table 2 illustrates the strong performance of the proposed algorithm (shown in bold): it achieves 
almost the same scores (an MAE of 1.04 and an RMSE of 2.33) as the best results in the literature. In the 
second test, the Comma.ai benchmark was used to evaluate the performance of the proposed method. Table 3 
shows a comparison of the proposed system with other schemes on the same dataset. It is clear from Table 3 
that the proposed method outperforms other methods in both the MAE and RMSE evaluation metrics. New 
state-of-the-art scores were achieved: an MAE of 0.64 and an RMSE of 0.87. 


Table 3. Comparison of the proposed algorithm with existing methods on the Comma.ai dataset 
Method [ref.] MAE (degrees) | RMSE (degrees) 


FM-Net [14] 0.7 0.98 
CNN + FCN [16] 2.54 3.26 
CNN + LSTM [16] 2.58 3.39 

Chen [18] 2.42 - 
Ours 0.64 0.87 


3.3. Discussion 

The proposed method was tested on two standard benchmarks, Udacity and Comma.ai. In the first test 
results listed in Table 2, our model shows almost the same results as [13]. Their scores were slightly above 
ours due to the additional processing stages they used: their approach connects several networks together, each 
of which performs its task and feeds the processed data to a “super” network that learns the important features 
from each network. This method has a high computational cost compared with our algorithm. Still, our method 
achieves more than 98% accuracy, which is close to the best-registered results for the Udacity dataset. In the 
second experiment, the Comma.ai dataset was used to test and evaluate the proposed system. The results in 
Table 3 demonstrate the superiority of the proposed method over other approaches on the same dataset: our 
algorithm outperformed the best-registered results by almost 9% and 11% for MAE and RMSE, respectively. 

However, the accuracy values presented in this and previous studies are imperfect. Both evaluation 
metrics (MAE and RMSE) depend on the difference between the predicted steering angle and the recorded 
ground-truth angle. This ground-truth angle represents the driving style of a specific human driver and is not 
necessarily the optimal steering angle at that time. It is also not possible for the driver to keep the car precisely 


Steering angle prediction via neural networks (Fayez Saeed Faizi) 


398 0 ISSN: 2502-4752 


in the middle of the lane for the entire time they are driving. This means that the predicted steering angle does 
not have to be exactly the same as the value attained by a human driver, because the same driver may record 
different steering angle values when driving again on the same road. So, one can say that as long as the car 
stays in the lane and follows its path, the current steering angle can be considered a correct angle. 


4. CONCLUSION 

This work investigates steering angle prediction for an autonomous vehicle as a regression problem 
and solves it using an ANN. The proposed prediction algorithm obtains data that represent the coordinates of 
some points on the lane line and then checks if one or both lines were detected in the previous stage. Based on 
that, the system determines the equation of the path line. Finally, the coefficients of the evaluated equation are 
fed into the trained NN, which predicts the steering angle for that frame. The experimental results, from testing 
on two different benchmarks, demonstrated the accuracy and robustness of the proposed algorithm and its 
superiority over other state-of-the-art NN-based methods. Nevertheless, the accuracy of the scheme is affected 
by the accuracy of the lane detection algorithm and its ability to extract the required features efficiently. 
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